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PREFACE 


As  a  result  of  the  1981  Defense  Science  Board  Summer  Study' 
on  Operational  Readiness,  Task  Order  T-2-126  was  generated  to 
look  at  potential  steps  toward  improving  the  Materiel  Readiness 
Posture  of  DoD  (short  title:  R&M  Study).  This  task  order  was 
structured  to  address  the  improvement  of  R&M  and  readiness 
through  innovative  program  structuring  and  applications  of  new 
and  advancing  technology.  Volume  I  summarizes  the  total  study 
activity.  Volume  II  integrates  analysis  relative  to  Volume  III, 
program  structuring  aspects,  and  Volume  IV,  new  and  advancing 
technology  aspects. 

The  objective  of  this  study  as  defined  by  the  task  order 
is : 


"To  identify  and  provide  support  for  high  payoff  actions 
which  the  DoD  can  take  to  improve  the  military  system 
design,  development  and  support  process  so  as  to  pro¬ 
vide  quantum  improvement  in  R&M  and  readiness  through 
innovative  uses  of  advancing  technology  and  program 
structure. " 

The  scope  of  this  study  as  defined  by  the  task  order  is: 

To  (1)  identify  high-payoff  areas  where  the  DoD  could 
improve  current  system  design,  development  program 
structure  and  system  support  policies,  with  the  objective 
of  enhancing  peacetime  availability  of  major  weapons 
systems  and  the  potential  to  make  a  rapid  transition  to 
high  wartime  activity  rates,  to  sustain  such  rates  and  to 
do  so  with  the  most  economical  use  of  scarce  resources 
possible,  (2)  assess  the  impact  of  advancing  technology 
on  the  recommended  approaches  and  guidelines,  and  (3) 
evaluate  the  potential  and  recommend  strategies  that 
might  result  in  quantum  increases  in  R&M  through  inno¬ 
vative  uses  of  advancing  technology. 
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The  approach  taken  for  the  study  was  focused  on  producing 
meaningful  implementable  recommendations  substantiated  by  quan¬ 
titative  data  with  implementation  plans  and  vehicles  to  be  pro¬ 
vided  where  practical.  To  accomplish  this,  emphasis  was  placed 
upon  the  elucidation  and  integration  of  the  expert  knowledge 
and  experience  of  engineers,  developers,  managers,  testers  and 
users  involved  with  the  complete  acquisition  cycle  of  weapons 
systems  programs,  as  well  as  upon  supporting  analysis.  A  search 
was  conducted  through  major  industrial  companies,  a  director 
was  selected,  and  the  following  general  plan  was  adopted. 

GENERAL  STUDY  PLAN 

Vol .  Ill  •  Select,  analyze  and  review  existing 
successful  program 

Vol.  IV  •  Analyze  and  review  related  new  and 
advanced  technology 

Vol.  II  (  •  Analyze  and  integrate  review  results 
(  •  Develop,  coordinate  and  refine  new 
concepts 

Vol.  I  •  Present  new  concepts  to  DoD  with  imple¬ 

mentation  plan  and  recommendations  for 
application. 

The  approach  to  implementing  the  plan  was  based  on  an 
executive  council  core  group  for  organization,  analysis,  inte¬ 
gration  and  continuity;  making  extensive  use  of  working  groups, 
heavy  military  and  industry  involvement  and  participation,  and 
coordination  and  refinement  through  joint  industry/service 
analysis  and  review.  Overall  study  organization  is  shown  in 
Fig.  P-1. 

The  basic  case  study  approach  was  to  build  a  foundation 
for  analysis  and  to  analyze  the  front-end  process  of  program 
structuring  for  ways  to  attain  R&M ,  mature  it,  and  improve  it. 
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FIGURE  P-1.  Study  organization 


Concurrency  and  resource  implications  were  considered.  Tools 
to  be  used  to  accomplish  this  were  existing  case  study  reports, 
new  case  studies  conducted  specifically  to  document  quantitative 
data  for  cross-program  analysis,  and  documents,  presentations, 
and  other  available  literature. 

This  document  records  the  R&M  program  review  elements 
which  were  used  as  guidelines  for  the  case  study  activities. 

The  views  expressed  within  this  document  are  those  of  the 
working  group  only.  Publication  of  this  document  does  not 
indicate  endorsement  by  IDA,  its  staff,  or  its  sponsoring 
agencies. 
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CONTRACT 


ELEMENT:  R&M  Requirements 

DEFINITION :  This  element  addresses  the  question  of  whether  the 

reliability  f maintainability  or  diagnostic)  re  irements  are  in¬ 
cluded  in  the  contract  as  contractual  require',  its  or  as  goals. 

SCOPE:  This  element  will  address  reliability,  maintainability 

and  testability.  It  will  address  both  quantitative  requirements 
and  qualitative  requirements. 

QUESTIONS  TO  BE  ADDRESSED:  Were  there  quantitative  and  qualita¬ 
tive  R&M  requirements?  Are  they  stated  clearly?  is  their 
growth  to  a  mature,  or  field  value  specified?  Were  requirements 
sufficiently  detailed  and  were  verifications  also  required? 

How  do  these  compare  to  similar  systems?  List  the  requirements 
together  with  assessment  of  difficulty  in  attaining. 

DISCUSSION :  A  number  of  years  ago,  R&M  programs  routinely 

included  quantitative  R&M  goals.  This  approach  placed  R&M  at 
a  disadvantage  versus  other  technical  requirements  on  the 
system  which  were  specified  as  contract  requirements.  Today 
most  programs  include  quantitative  R&M  requirements  if  they  are 
appropriate  for  the  item  being  developed.  To  properly  compete 
for  contractor  attention  and  resources,  the  R&M  numerics  should 
be  specified  in  the  contract  (specification  or  statement  of 
work)  as  contract  requirements  as  opposed  to  goals.  Qualitative 
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areas  may  also  be  specified  as  requirements.  All  contract 
requirements  must  have  a  contractually  specified  verification 
to  be  truly  enforceable  requirements.  There  are  several  measures 
of  R&M  which  qualify  the  numeric  value,  and  must  be  taken  into 
consideration,  such  as  inherent,  induced,  no  defect,  field 
removal  rates,  etc.  These  can  drive  design  decisions  and  must 
be  distinguished  from  each  other  in  documenting  the  requirements. 
The  planned  R&M  growth,  if  any,  must  also  be  considered  (see 


Test  and  Evaluation). 
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CONTRACT 


ELEMENT:  Mission  Profile  Establishment 


DEFINITION :  The  mission  profile  is  a  time-phased  description 

of  the  events  and  environments  an  item  experiences  from 
initiation  to  completion  of  a  specified  mission,  to  include 
the  criteria  of  mission  success  or  critical  failures. 


SCOPE:  This  element  should  address  the  development  of  the 

mission  profile  for  the  system  or  a  number  of  mission  profiles 
for  multi-mission  systems.  The  profile(s)  shall  define,  as  a 
minimum,  the  boundaries  of  the  performance  envelope  and  provide 
the  timeline  (environmental  conditions  and  applied/induced 
stresses  versus  time)  typical  of  operations  within  that  envelope 
(e.g.,  engine  throttle  cycles). 


QUESTIONS  TO  BE  ADDRESSED:  Was  a  mission  profile  (or  profiles) 
defined  prior  to  full  scale  development  or  earlier?  Did  the 
profile  address  in  adequate  detail  the  time  phasing  of  operation, 
maintenance,  the  predicated  environmental  conditions,  and  the 
rate  of  change  of  system  conditions?  Were  these  mission  profile 
conditions  and  operations  used  as  inputs  to  the  stress  and 
design  analyses?  To  what  level  of  system  partitioning  was  the 
profile  apportioned?  Did  the  system  design  criteria,  worst 
case  analysis,  derating  criteria  BIT/FIT  compensate  for  the 
conditions  identified  in  the  mission  profile?  Was  the  defined 
mission  profile  evaluated  during  early  system  testing  and 
operation  to  determine  if  it  was  still  valid?  Were  there  changes 
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to  the  profile?  When?  How  were  these  identified  to  the  contrac 
tor? 


DISCUSSION :  Prior  to  the  beginning  of  detailed  design,  a 

mission  profile  (or  profiles)  should  be  defined  to  indicate  the 
system  utilization,  maintenance  concept,  time  phase  sequence 
of  system  operation,  and  time  phase  sequence  and  period  of 
environmental  conditions.  This  profile  must  be  based  on  the 
best  available  data,  but  it  must  be  recognized  that  predicting 


system  usage  and  environments  is  not  an  exact  science.  As  the 
system  is  developed,  additional  effort  must  be  expended  to 
determine  if  these  projections  have  proved  to  be  incorrect. 


112/5-2 


CONTRACT 


ELEMENT:  Life  Profile  Establishment 


DEFINITION :  The  life  profile  is  a  time-phased  description  of 

the  events  and  environments  an  item  experiences  from  manufacture 
to  final  expenditure  or  removal  from  the  operational  inventory, 
to  include  one  or  more  mission  profiles. 


SCOPE :  This  element  addresses  the  entire  life  profile,  with 
the  exception  of  the  details  of  the  mission  profile(s)  which 
are  addressed  in  the  mission  profile  element. 


QUESTIONS  TO  BE  ADDRESSED:  Has  the  life  profile  (production 
through  disposal)  been  established?  Does  the  system  have 
periods  of  non-mission  time  that  are  significant  to  the  develop¬ 
ment  because  of  their  relative  duration  (e.g.,  missiles  with 
long  storage  periods)  or  unusual  environments  (e.g.,  electronics 
stored  in  an  uncontrolled  environment)?  What  is  the  basis  for 
estimates  of  the  environment  to  be  expected  in  these  non-mission 
phases?  Does  this  profile  address  the  impact  of  humans  on  the 
system  in  the  phases  of  the  life  profile?  How  did  the  life  profile 
impact  on  the  detailed  design  requirements?  Were  specific  design 
or  support  alternatives  considered?  Did  the  program  validate 
the  environments  of  the  life  profile  through  the  test  program 
or  in-service  assessments? 
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DISCUSSION :  During  the  development  of  system  requirements 

it  is  essential  to  develop  a  definition  of  the  system’s  life 
profile.  This  is  particularly  true  if  some  of  the  unique 
aspects  of  the  non-mission  portions  of  the  life  profile  need 
to  be  translated  into  design  requirements.  The  definition  of 
the  life  profile  needs  to  address  operational  and  support  phases 
that  may  not  be  accurately  defined  by  the  system  contractor. 

It  is  a  question  of  identifying  what  is  actually  done  in  opera¬ 
tional  service  versus  information  contained  in  planning  documents. 
There  is  also  a  potential  problem  in  that  new  systems  may 
cause  user  personnel  to  change  their  method  of  operation  and 
to  cause  a  significant  change  in  the  life  profile  without 
adequate  evaluation  of  the  technical  impact. 


CONTRACT 


ELEMENT:  R&M  Failure  Definition 

DEFINITION:  This  element  is  the  definition  of  failure, 

malfunction,  maintenance,  etc.,  that  provide  meaning  to  other 
elements  such  as  requirements  and  testing. 

SCOPE:  This  element  addresses  not  only  the  definition  of 

failures,  but  also  other  definitions  that  pertain  to  R&M 
activities  including  maintenance,  man  hours,  diagnostic 
detections,  no  defect  removal  rates,  downtime,  etc. 

QUESTIONS  TO  BE  ADDRESSED:  Was  the  definition  of  failure 
established  prior  to  design  and  in  concert  with  the  R&M 
requirements?  Was  there  an  operational  definition  in  addition 
to  a  contract  definition?  Was  the  definition  adequate  to 
prevent  conflict  during  testing  and  adequate  to  result  in  R&M 
results  that  were  meaningful?  Were  the  maintainability  and 
diagnostic  requirements  adequately  supported  by  definitions? 

DISCUSSION :  The  R&M  requirements  must  be  established  in  a 

defined  context.  This  definition  of  failure,  manhours, 
maintenance,  operating  time,  critical  failures,  etc.,  must  be 
completed  before  the  quantitative  R&M  requirements  are  established 
and  prior  to  detailed  design  activity  if  the  R&M  requirements 
are  to  have  any  impact  on  the  design.  Different  definitions 
should  be  used  to  define  operationally  relevant  conditions  and 
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to  define  contract  chargeable  conditions.  The  priority  should 
be  to  define  these  two  sets  accurately  for  their  two  distinct 
purposes  rather  than  compromising  the  separate  accurate 
definitions  for  a  single  less  accurate  set. 
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CONTRACT 


ELEMENT:  Incentives 


DEFINITION :  This  element  addresses  incentive  provisions  in  the 

contract  which  are  intended  to  motivate  contractor  actions  which 
will  result  in  a  more  reliable  (or  maintainable)  product. 


SCOPE:  This  element  includes  R&M  incentives,  warranty  and 

award  fee  provisions  that  are  based  upon  equipment  performance 
or  contractor  performance  with  the  objective  of  improving  the 
reliability  and/or  the  maintainability  of  the  item. 

QUESTIONS  TO  BE  ADDRESSED:  Which  type  (or  combination  of  types) 
(warranty,  award  fee,  incentive  provisions)  is  being  used?  Are 
the  incentives  based  upon  contractor  performance  or  product 
(i.e.,  equipment,  system)  performance.  If  product  performance, 
what  is  the  basis  measure  and  measurement  period?  How  realistic 
are  they  when  compared  to  operational  conditions?  How  do  the 
R&M  incentives  compare  to  the  total  incentive  program  on  the 
contract  (relative  weight,  dollar  amounts,  time  phase,  etc.). 

Are  the  incentives  and  the  basis  measurement  periods  consistent 
with  the  program  schedule? 


DISCUSSION :  The  R&M  incentive  approach  to  a  particular 

program  should  be  based  upon  the  critical  R&M  parameters  or 
characteristics  for  that  system.  Once  the  critical  parameters 
have  been  identified  then  the  various  incentive  approaches  must 
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be  evaluated.  Some  approaches  have  their  basis  measurement  only 
in  operational  service  (e.g.,  reliability  improvement  warranties) 
Others  may  be  applied  in  any  time  phase  (e.g.,  award  fee).  The 
correct  approach  is  to  select  the  right  combinations  of  critical 
parameters  and  incentives  approaches.  The  R&M  incentives  must 
also  be  viewed  in  relation  to  the  total  incentive  package  on  the 
contract  to  assure  proper  relative  weight  is  given  to  R&M. 
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CONTRACT 


ELEMENT:  Source  Selection  Criteria 

V\ 

DEFINITION :  This  element  addresses  the  question  of  how  the 

source  selection  criteria  aid  in  contributing  to  the  R&M  objec¬ 
tives. 

SCOPE :  This  element  considers  both  the  customer's  evaluation 

criteria  stated  in  the  RFP  and  the  weighting  or  scoring  method 
used  by  the  source  selection  evaluation  board  (SSEB).  (This 
latter  material  may  be  difficult  to  obtain  and/or  evaluate 
because  of  the  "source  selection  sensitive"  nature.) 

QUESTIONS  TO  BE  ADDRESSED:  How  was  R  and/or  M  addressed  in  the 
RFP  statement  of  evaluation  criteria?  (Recognize  that  while 
the  SSEB  scoring  may  be  more  definitive,  the  RFP  is  visible  to 
O  the  contractors  and  has  the  effect  of  communicating  the  govern¬ 

ment's  relative  priority  on  the  various  factors.)  Are  there 
confusing  or  conflicting  priorities  between  the  RFP  statement, 
relative  incentive  amounts,  or  other  government  statements? 

(One  potential  source  of  conflict  may  be  the  RFP  cover  letter.) 


DISCUSSION :  The  government's  priority  on  R&M  must  be 

reflected  in  both  the  RFP  statement  of  evaluation  criteria  and 
in  the  SSEB  weighting  method.  The  first  of  these  can  motivate 
the  contractor  to  propose  his  best  design  or  approach  to  the 
R&M  problem.  The  latter  ensures  the  proper  evaluation  of  the 
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alternative  R&M  characteristics  and  the  potential  for  selecting 
the  best  approach  from  an  R&M  standpoint.  Contractors  will  not 
attach  high  value  to  R&M  until  they  see  the  government  making 
selection  of  designs  with  good  R&M  features  and  rejecting  ones 
with  poor  R&M  potential. 


CONTRACT 


ELEMENT:  LCC  Consideration 


DEFINITION :  Life  cycle  cost  consideration  is  the  situation 

where  the  R&M  requirements  are  developed  considering  life 
cycle  cost  and  how  it  is  impacted  by  R&M . 


SCOPE:  The  element  includes  how  LCC  considerations  impacted  on 

the  R&M  design  related  requirements  development.  It  also 
includes  the  impact  on  R&M  of  LCC  requirements,  LCC  contract 
incentives  and  other  contract  requirements. 

QUESTIONS  TO  BE  ADDRESSED:  Was  life  cycle  cost  a  factor  in  the 
development  of  R&M  design  related  requirements  and  design  trade¬ 
offs?  What  cost  factor(s)  (total  LCC,  operations  and  support 
cost,  just  support  cost)  was  used?  How  was  it  done?  What  speci 
fication  was  used?  How  did  the  requirement  satisfy  both  LCC 
and  mission  requirements?  Was  LCC  estimated  and  evaluated  as 
the  program  progressed?  Did  this  information  impact  R&M  activi¬ 
ties?  Was  there  insight  into  the  LCC  implications  of  identified 
R&M  shortfalls?  Was  LCC  re-evaluated  after  the  test  and  evalua¬ 
tion  program  or  after  production  began? 


DISCUSSION :  One  of  the  principle  benefits  of  good  levels  of 

R&M  is  to  reduce  life  cycle  cost.  Therefore  LCC  analysis 
should,  either  directly  or  indirectly,  be  part  of  the  process 
to  determine  the  R&M  requirements.  During  design  trade-off 
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studies,  LCC  should  be  addressed  as  one  factor  in  evaluating 
alternatives.  During  T&E  and  early  deployment,  R&M  problems 
might  be  assessed  against  LCC  impact.  LCC  might  also  be  asses¬ 
sed  to  evaluate  the  relative  impact  of  system  characteristics, 
including  R&M.  Payoff  of  this  item  is  usually  subject  to 
skepticism  due  to  the  inherent  difficulty  in  accurately  modeling 
cost  impact  of  the  individual  elements  to  the  total  cost  (due 
to  lack  of  credible  and  sufficient  data,  analytical  models, 
consistency,  etc.). 


MANAGEMENT 


ELEMENT:  Planning,  Control  and  Emphasis 

DEFINITION:  This  element  consists  of  both  government  and  con¬ 

tractor  management  of  the  effort  to  produce  a  reliable  and 
maintainable  system. 


SCOPE:  This  element  will  include  all  of  the  PM's  management 

actions  that  are  part  of  an  R&M  progam.  It  will  specifically 
address  R&M  emphasis,  planning  and  control. 


QUESTIONS  TO  BE  ADDRESSED:  Is  the  emphasis  placed  on  R&M  by 
the  government  and/or  contractor  program  manager  visible  and 
accessible?  Were  the  PMs  involved  in  the  major  R&M  reviews  or 
was  it  left  to  the  R&M  staffs?  Were  there  senior  management 
communiques  or  meetings  on  the  subject  of  the  R&M  program? 

Were  the  program  plans  adequate,  comprehensive,  or  perfunctory? 
What  evidence  is  there  that  these  were  adhered  to?  Were  the 
R&M  tasks  scheduled  in  a  manner  consistent  with  the  planned 
design  schedule?  Was  the  schedule  compressed  or  concurrent? 

How  did  the  program  actually  function  as  compared  to  the  schedule 
What  records  were  kept  for  management  visibility?  How  did  the 
actual  conduct  of  R&M  tasks  relate  to  the  timing  of  design 
activity?  Were  there  any  financial  penalties  or  incentives  on 
the  timely  conduct  of  program  tasks? 


DISCUSSION ;  The  environment  that  is  established  by  senior 
management  (government  and  contractor)  can  be  critical  to  the 
success  of  the  R&M  program.  This  emphasis  is  communicated 
through  personal  involvement,  written  comments,  and  participa¬ 
tion  in  major  R&M  and  design  meetings.  The  planning  for  R&M 
activities  must  be  thorough  but  flexible.  It  must  be  consistent 
with  the  schedule  of  design  activity  so  that  the  products  of 
R&M  activities  can  influence  the  design  rather  than  after-the- 
fact  documentation.  Management  also  requires  controls.  There 
should  be  visibility  and  control  procedures  to  assure  that  the 
R&M  tasks,  including  those  of  the  subcontractors,  are  being 
accomplished  correctly  and  on-time. 
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MANAGEMENT 


ELEMENT:  Monitor/Control  of  Subcontractors  and  Suppliers 

DEFINITION :  This  element  consists  of  management  and  contract 

methods  between  the  prime  contractor  and  his  subcontractors  and 
suppliers . 

SCOPE :  This  element  includes  R&M  contract  arrangements  and 

management  emphasis  on  R&M  between  the  prime  contractor  and  his 
subcontractors  and  suppliers. 

QUESTIONS  TO  BE  ADDRESSED:  Have  the  allocated  R&M  requirements 
been  passed  on?  How  were  these  tailored  to  the  subcontractor? 

How  did  the  prime  contractor  monitor  subcontractor  performance 
during  design,  analysis,  etc?  Was  subcontractor  R&M  emphasis 
judged  as  adequate?  How  much  contractor  R&M  activity  was 
directed  at  internal  design  versus  subcontractors  or  suppliers? 
Were  any  unusual  methods  used  to  communicate  the  priority  on 
R&M  to  the  subcontractors  and  suppliers?  Were  R&M  incentives 
passed  on  to  the  subcontractor?  How  were  the  subcontractors 
and  suppliers  integrated  into  the  other  reliability  tasks  such 
as  part  and  material  control  and  failure  analysis/corrective 
action? 

DISCUSSION :  On  many  systems,  the  R&M  critical  components  are 

subcontracted.  The  subcontracts  must  reflect  the  R&M  requirements 
and  philosophy  in  the  prime  contract.  The  incentives  and  the 


management  priority  placed  on  R&M  by  the  government  must  be 
accurately  and  completely  communicated  to  the  subcontractors 
The  subcontractors  *  role  in  the  R&M  program  can  be  critical. 
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ELEMENT:  Development  of  Design  Requirements 


DEFINITION :  Development  of  design  requirements  is  the  process 

that  translates  program  goals  and  thresholds  to  requirements 
for  contractual  specifications,  statements  of  work,  and  for 
communication  to  the  designers. 

SCOPE :  This  element  includes  the  analysis  to  establish  the 

relationship  between  the  R&M  requirements  and  other  readiness- 
related  parameters,  identify  chargeable  R&M  requirements, 
develop  and  allocate  contract  R&M  requirements,  and  translate 
R&M  requirements  for  the  designers  in  terms  of  design  attributes. 

QUESTIONS  TO  BE  ADDRESSED:  Were  the  program  R&M  goals  and 
thresholds  developed  as  operational  measures  (as  opposed  to 
contract  chargeable  measures)?  Are  they  consistent  with  other 
readiness-related  goals  and  thresholds  (i.e.,  availability)? 

How  were  the  contract  R&M  requirements  developed?  How  did  this 
process  account  for  scenario,  mission  mix,  wartime/peacetime, 
etc?  Was  a  baseline  system  developed  from  previous  systems? 

To  what  level  were  R&M  requirements  allocated?  Was  there  an 
assessment  of  technical  risk  in  the  R&M  allocation?  How  were 
R&M  requirements  translated  to  the  design  engineer? 

DISCUSSION :  The  program  R&M  goals  and  thresholds  should  be 

developed  based  on  the  user's  statement  of  his  required 


capability.  This  must  be  based  on  defined  utilization,  missio 
mix,  wartime/peacetime,  etc.  A  baseline  should  be  defined 
using  data  on  previous  systems.  From  the  goals  and  thresholds 
chargeable  R&M  requirements  should  be  developed  that  reduce  th 
area  of  interest  to  those  things  that  are  chargeable  to  the 
contractor ( s ) .  There  should  be  consistency  between  the  R&M 
parameter  values  and  other  readiness-related  goals,  thresholds 
and  requirements.  The  allocation  of  system  requirements  to 


DESIGN 


ELEMENT:  Design  Alternative  Studies 


DEFINITION:  This  element  includes  all  of  the  trade  studies  or 

studies  of  alternatives  that  are  accomplished  prior  to  and  in 
the  early  stages  of  design. 


SCOPE :  The  scope  of  this  element  is  only  those  studies  that 

examine  design  alternatives  that  may  result  in  changes  in 
reliability  and/or  maintainability.  These  studies  might  include: 
simplification,  inspectability ,  producibility ,  redundancy, 
useful  life,  LCC,  BIT  vs.  ATE,  partitioning,  etc. 


QUESTIONS  TO  BE  ADDRESSED:  Were  design  trade  studies  conducted 
aimed  at  improving  R&M?  Were  other  studies  evaluated  for  impact 
on  R&M?  Was  this  trade  study  activity  directed  in  some  manner 
or  was  it  open  ended?  How  many  trade  studies  were  performed? 

How  (or  by  whom)  were  the  results  evaluated?  How  were  they 
documented?  Were  the  study  results  acted  upon?  Were  the 
studies  used  to  identify  R&M  improvements  or  were  they  more 
correctly  used  to  identify  deficiencies  or  problem  areas?  Was 
the  degree  of  freedom  to  change  specified?  If  so,  how  and  where? 
Did  the  trades  result  in  identifiable  cost  savings  to  the  con¬ 
tractor? 


DISCUSSION :  The  aggressive  conduct  and  acceptance  of  alternative 

design  trade  studies  can  have  a  significant  impact  in  establishing 

* 
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the  design  (and  R&M  of  the  design)  baseline.  If  the  trade 
study  program  is  aggressively  managed,  it  will  be  directed  at 
finding  a  proper  balance  between  the  many  demands  on  the  system 
design.  This  balance  should  attach  enough  importance  to  readi¬ 
ness  and  support  considerations  and  risk  that  alternatives 
also  examine  improving  R&M  to  the  detriment  of  other  system 
characteristics.  While  trade  studies  may  uncover  design  defi¬ 
ciencies,  the  thrust  of  the  program  should  be  directed  at 
developing  improvements  in  the  baseline  design.  Trades  may  also 
result  in  changing  support  concepts  or  deployment  to  reduce  the 
support  tail,  but  this  study  is  directed  toward  the  equipment 
design  aspects. 


DESIGN 


ELEMENT:  Design  Evaluation  Analysis 

DEFINITION:  This  element  is  the  analyses  that  evaluate  the 

design  and  provide  a  basis  for  iterative  design  influence, 
correction  or  improvement  from  an  R&M  standpoint. 


SCOPE:  This  element  includes  thermal  (and  other  environmental) 

analysis,  electrical  (and  other)  stress  analysis,  worst  case 
tolerance  analysis,  sneak  circuit  analysis  and  other  detailed 
analyses  that  should  be  done  as  the  detailed  design  is  done  on 
an  iterative  basis,  numerical  predictions,  failure  modes, 
effects  and  criticality  analysis,  thermal  survey,  and  maintaina¬ 
bility  analysis. 


QUESTIONS  TO  BE  ADDRESSED:  Were  these  analyses  integral  parts 
of  the  design  process?  Who  performed  these  analyses?  Were 
they  contractually  required  or  a  contractor  initiative?  Did 
the  results  of  these  analyses  have  an  impact  on  the  design? 
What  evidence  is  there  of  this?  How  did  management  view  these 
tasks  (i.e.,  work  to  be  done,  or  inputs  to  design  or  merely 
contractual  deliverables)?  As  the  detailed  design  was  changed 
during  the  development  program,  were  the  R&M  design  analyses 
updated  to  evaluate  the  design  changes?  If  allowed,  would  the 
contractor  reduce  these  requirements? 
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DISCUSSION :  The  design  evaluation  analyses  are  best  done 

by  designers  as  an  integral  part  of  the  detailed  design.  The 
results  of  these  analyses  must  provide  information  to  designers 
and  engineering  managers  so  the  analyses  are  actually  causing 
change  in  the  design  rather  than  just  recording  the  design  that 
exists.  The  timing  and  the  credibility  of  these  analyses  must 
be  such  that  they  are  accepted  as  part  of  the  design  evolution. 
A  common  flaw  in  many  R&M  programs  is  to  have  the  R&M  analyses 
separated  from  design  activity  by  time  (in  this  situation  the 
analysis  merely  documents  the  characteristics  of  the  design 
without  affecting  the  design  chosen),  distance  or  organization. 
Separation  by  organization  can  have  positive  or  negative  impact 
depending  upon  the  organizational  factors  (and  personalities) 
involved.  These  situations  require  evaluation  that  may  be 
difficult  in  a  short  period  of  time.  The  role  of  R&M  analyses 
in  design  changes  can  often  be  an  indicator  of  the  importance 
placed  upon  these  tasks  by  the  design  team. 
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DESIGN 


ELEMENT:  Parts  and  Material  Selection  and  Control 


DEFINITION :  This  element  includes  the  procedure,  process  and 

controls  used  by  the  government  and  the  contractor  to  assure 
that  the  parts  and  materials  used  to  manufacture  the  item  are 
proper  for  the  required  level  of  R&M . 


SCOPE :  This  element  addresses  both  parts  and  material  selection 

It  includes  environmental  stress  screening  requirements  imposed 
at  the  part  vendor's  facility. 


QUESTIONS  TO  BE  ADDRESSED:  Did  the  program  follow  the  parts 
control  process  in  MIL-STD-965  (or  its  predecessor  documents)? 

Was  there  a  parts  control  board  and/or  an  approved  parts  list? 

Were  electronic  parts  restricted  to  the  "high  reliability" 
specifications  (ER,  TX,  TXV,  38510,  etc.)?  Who  could  grant  an 
exception  to  this  rule?  Was  there  source  control  of  standard 
and  non-standard  parts?  What  numerical  percent  of  the  electronic 
parts  were  from  the  approved  "high  reliability"  specifications? 

How  were  materials  controlled?  Was  there  a  board  to  control  the 
material  selection?  Were  the  parts  and  material  control  procedures 
contractual?  What  control  was  exercised  by  the  government?  Did 
this  element  have  an  identifiable  cost  (in  dollars  and/or  man¬ 
hours)?  Were  there  identifiable  benefits  to  the  contractor  above 
meeting  contractual  requirements  (i.e.,  did  assembly  test  reject 
rate  improve)? 
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DISCUSSION:  A  well  defined,  properly  executed  parts  (and 

material)  selection  program  can  reduce  one  source  of  potential 
reliability  problems.  Restricting  appropriate  classes  of 
parts  to  well-understood,  high  volume  components  with  increased 
assurance  of  part  quality,  can  contribute  significantly  to 
producing  a  reliable  product.  The  control  procedures  should  be 
well-defined.  Part  screening  requirements  should  be  established 
and  enforced.  The  approved  parts  list  should  be  available 
before  the  designer  is  ready  to  select  components.  The  authority 
for  granting  exceptions  to  the  control  procedure  should  be  well- 
defined  and  monitored  to  assure  that  exceptions  are  not  granted 
too  freely.  This  element  must  have  a  close  interaction  with  the 
environmental  stress  screening  of  parts,  assemblies,  etc. 
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DESIGN 


ELEMENT:  Derating  Criteria 


DEFINITION :  This  element  addresses  the  establishment  of  criteria 

for  acceptable  derating  and  the  enforcement  of  that  criteria. 


SCOPE :  The  element  addresses  derating  of  component  parts  and 

materials . 


QUESTIONS  TO  BE  ANSWERED:  Was  there  an  established  derating 
criteria?  Who  established  it?  Who  could  approve  deviations? 
What  magnitude  of  deviations  were  approved?  Were  these 
significant  in  the  context  of  the  total  design?  How  did  the 
criteria  compare  with  other  existing  standards  (e.g.,  NAVSEA, 
NAVMAT,  RADC  TR)?  How  did  the  resulting  design  compare  with 
the  derating  criteria?  Were  measurements  taken  of  actual 
stress  levels?  What  were  the  highest  stress  level  parts  and 
what  technical  requirements  (e.g.,  weight,  size)  impacted  the 
derating  levels?  Did  failure  data  indicate  that  the  criteria 
was  acceptable  and  complied  with?  Are  thermal  evaluations  made 
in  the  field  and  empirical  data  gathered,  if  required,  and 
compared  to  predicted  derating? 


DISCUSSION:  Almost  all  programs  have  derating  criteria. 

The  principal  problem  has  been  enforcement  or  the  granting  of 
deviations.  Derating  criteria  should  be  established  consistent 
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with  one  of  the  published  guidelines.  Careful  enforcement  should 
assure  that  unnecessary  deviations  are  not  granted. 
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DESIGN 


ELEMENT:  Thermal-Packaging  Criteria 

DEFINITION :  This  element  consists  of  the  design  criteria  and 

guidelines  that  are  established  to  guide  designers  to  provide  a 
design  that  has  proper  thermal  and  packaging  characteristics. 

SCOPE:  This  criteria  addresses  the  thermal  and  packaging 

characteristics  of  equipment  (primarily  electronics  although 
other  sensitive  items  should  be  included)  and  other  environmental 
characteristics  that  might  be  critical  to  the  design,  e.g., 
vibration. 

QUESTIONS  TO  BE  ADDRESSED:  Was  a  criteria  established  for  the 
thermal  design?  Was  this  criteria  influenced  by  the  contract? 

Was  it  consistent  with  the  derating  criteria?  Was  implementation 
of  the  criteria  monitored  by  the  design  reviews?  Were  there 
tests  performed  to  verify  the  implementation  of  the  criteria 
and  to  determine  the  actual  pay-off  in  equipment  characteristics 
(junction  temperatures)?  How  was  the  packaging  criteria 
addressed? 

DISCUSSION:  The  approach  and  criteria  of  thermal  and  packaging 

design  can  be  critical  to  the  success  of  the  design.  The 
criteria  might  address  the  thermal  paths  to  be  used  (e.g., 
direct  cooling,  cold  walls,  flow  through  cold  plate  cooling). 
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Reliability  Engineers."  This  document  provides  a  good  review 
of  this  subject  area. 


DESIGN 


ELEMENT;  Computer  Aided  Design 


DEFINITION ;  This  element  consists  of  computer  aided  engineering 
tools  to  help  design  a  reliable  and  maintainable  product. 


SCOPE ;  This  element  includes  computer  assisted  analysis  that 
aids  in  assessing  the  R&M  of  the  design  or  optimizing  the  R&M 
of  the  design.  This  includes  computer-aided  circuit  development 
thermal  and/or  vibration  analysis,  R  or  M  prediction,  circuit 
board  layout,  test  point  selection,  test  and  troubleshooting 
programming  and  partitioning. 

QUESTIONS  TO  BE  ADDRESSED;  Were  computer  aided  R&M  tools  used? 
Which  ones?  How  did  these  analysis  tools  and  their  products 
interrelate  with  the  main  detailed  design  process?  Did  these 
tools  make  it  easier  to  create  a  reliable  design  and  reduce 
the  introduction  of  design  flaws  that  could  have  resulted  in 
lower  R&M?  Were  the  computer  tools  linked  together?  Were 
there  other  computer-aided-design  tools  used?  Did  these  tools 
optimize  the  design  on  a  basis  contrary  to  R&M?  What  are  the 
estimated  costs  vs.  benefits  compared  to  a  non-CAD? 


DISCUSSION;  Computer  aided  design  (CAD)  is  the  use  of  computer- 
assistance  to  help  the  designer  produce  a  design  that  satisfies 
a  number  of  design  constraints  and  optimizes  other  design 
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features.  Computer  analyses  are  tools  that  in  an  automated 
manner  attempt  to  optimize  a  characteristic  of  the  design,  or 
other  automated  analyses.  CAD  for  R&M  can  consist  of  R&M 
predictions  that  the  designer  can  evaluate,  the  identification 
of  overstressing,  sneak  paths,  malfunction  choke  points  as 

well  as  automated  decision  making  that  forces  the  design  alter¬ 
natives  to  meet  certain  constraints.  Key  to  evaluating  CAD  is 

to  examine  the  priority  of  R&M  in  the  CAD  process  as  compared 
to  technical  performance,  weight,  etc.  and  whether  viable, 
cost  effective  recommendations  resulted  from  the  process. 

Does  the  CAD  process  make  the  design  of  a  reliable,  maintainable 
product  harder  or  easier? 


DESIGN 


ELEMENT:  Testability  Analysis 


DEFINITION :  This  element  is  the  design  analysis  effort  related 

to  developing  the  diagnostic  approach  and  implementing  that 
approach . 


SCOPE :  This  element  includes  BIT  analyses,  nodal  analysis  for 

partitioning,  test  point  design,  fault  simulation,  and  diagnostic 
preparation  for  all  specified  levels  of  maintenance. 


QUESTIONS  TO  BE  ADDRESSED:  What  was  the  process  whereby  the 
contractor  defined  his  testability  (Built-In-Test)  design 
approach?  Was  the  testability  design  concurrent  with  or  after 
the  major  equipment  design?  Did  test  point  selection  and  design 
and  testability  partitioning  play  a  major  role  in  the  system 
layout  and  packaging?  Was  a  failure  modes  and  effects  analysis 
available  and  used  as  part  of  the  testability  analysis?  Were 
BIT  analyses  and  fault  simulation  used  to  evaluate  the  coverage 
and  effectiveness  of  the  BIT  design?  Was  the  testability 
design  approach  evolved  as  information  was  obtained  from  analyses 
and  test  experience?  How  did  the  results  compare  with  the  re¬ 
quirements? 

DISCUSSION :  Testability  analysis  should  be  used  to  define 

diagnostic  requirements  before  the  detailed  design.  During  the 


detailed  design,  test  point  analysis  and  nodal  analysis  for 
partitioning  should  be  major  inputs  to  the  layout/packaging 
design.  Analyses  of  projected  BIT  performance  and  fault 
simulation  studies  should  be  used  to  evaluate  the  progress 
toward  the  BIT  objectives  and  as  inputs  to  a  BIT  maturation 
program.  Design  improvement  data  should  also  be  extracted  from 
other  test  and  operational  data. 
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ELEMENT;  BIT  and  ATE  Performance 

DEFINITION:  This  element  is  the  task  associated  with  BIT/ATE 

requirements  development  and  BIT/ATE  evaluation  and  assessment. 

SCOPE:  This  element  includes  establishment  of  the  requirements 

for  the  diagnostics,  the  specification  of  these  requirements, 
the  assessment  of  BIT/ATE  performance,  and  the  maturation  of 
diagnostic  systems. 
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QUESTIONS  TO  BE  ADDRESSED:  What  are  the  BIT  requirements?  How 
were  they  derived?  What  is  the  relationship  between  the  BIT 
requirements  and  the  ATE?  Is  information  from  the  BIT  available 
to  and  used  by  the  ATE?  How  was  the  progress  of  the  BIT  design 
monitored  and  evaluated?  What  method  was  used  to  demonstrate, 
contractually,  the  BIT  performance?  How  valid  was  this 
prediction?  Were  any  unique  tools  or  methods  used  to  design, 
develop  or  mature  the  BIT  system?  How  was  the  BIT  system 
assessed  in  early  operation?  What  was  level  of  false  alarms, 
could  not  duplicates,  retest-OKs?  Were  there  BIT/ATE  compati¬ 
bility  problems? 

DISCUSSION :  The  procedure  to  define  the  required  BIT  performance 

has  not  been  developed  to  any  proven  level.  Regardless  of  the 
requirements,  the  analytical  assessment  of  BIT  levels  should 
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have  adequate  attention  to  assure  that  problems  are  corrected 
when  first  identified.  Similarly  there  must  be  attention  to 
early  test  and  early  operation  experience  with  BIT.  BIT/ATE 
compatibility  must  also  be  assured. 


DESIGN 


ELEMENT:  Features  to  Facilitate  Maintenance 


DEFINITION :  This  element  addresses  maintenance  requirements  and 

their  inclusion  into  design  constraints  directly  as  well  as  in¬ 
fluencing  R&M  requirements.  The  premise  is  that  maintenance 
requirements  must  be  supported  by  system  design. 


SCOPE :  This  element  includes  consideration  of  physical 

attributes,  diagnostic,  simplification,  repairabi 1 ity ,  testabil¬ 
ity,  inspectabi 1 ity ,  accessibility,  supportability  and  mainte¬ 
nance  time/cost  design  criteria  commensurate  with  the  mainte¬ 
nance  profile  of  the  system. 


QUESTIONS  TO  BE  ADDRESSED:  What  was  the  methodology  used  in 
developing  the  maintenance  concept?  Was  the  maintenance  concept 
systematically  reviewed?  How  was  it  specified?  Is  the  system 
design  compatible  with  the  maintenance  resource  development? 

Is  the  system  design  "friendly  to  maintenance"  in  providing 
accessibility,  fault  isolation  capability  and  packaging  for 
repairability?  Are  calibration  and  precision  measurements 
readily  accommodated?  Is  the  system  packaged  for  fault  isolation 
and  repairability  as  required  by  the  maintenance  concept, 
manpower  and  skill  restraints?  Can  sub-units  be  successfully 
maintained  at  their  designated  repair  levels  as  functional 
entities?  Are  design  trade-offs  considered  with  respect  to 
maintenance  requirements?  Are  trade-offs  documented,  reviewed 
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and  complied  with?  Is  there  on-going  maintenance  systems 
engineering  involvement  in  system  design? 

DISCUSSION :  Qualitative  and  quantitative  maintenance  requirements 

must  be  established  to  provide  system  design  guidelines  especially 
during  system  development.  Prior  to  detailed  design,  a  system 
maintenance  concept  should  be  formulated.  Maintenance  require¬ 
ments  identification  is  a  key  contributor  to  cost  effectiveness. 
When  Life  Cycle  Cost  is  minimized,  a  major  factor  is  optimum 
maintenance.  Improved  maintenance  enhances  system  availability, 
therefore  increases  user  operational  readiness.  The  ability  of 
maintenance  to  improve  operational  readiness  is  a  function  of  R&M . 
Maintenance  systems  engineering  develops  concepts,  criteria  and 
technical  requirements  to  assure  timely,  adequate  and  cost  effec¬ 
tive  support  of  system  design  and  operational  needs. 


MANUFACTURING 


ELEMENT:  Environmental  Stress  Screening  of  Parts/Equipment 

DEFINITION :  This  element  addresses  the  environmental  stress 

screening  (ESS)  of  parts,  assemblies,  and  systems  during  manu¬ 
facturing  . 


SCOPE:  This  element  includes  part  (or  component)  screening  as 

part  of  receiving  inspection,  screening  (or  non  operating 
"conditioning")  of  assemblies,  and  the  screening  of  "black  boxes 
subsystems  and  systems. 


QUESTIONS  TO  BE  ADDRESSED:  Has  an  integrated  stress  screening 
program  been  established?  How  were  the  requirements  established 
To  what  extent  is  the  contractor  doing  screening  to  incoming 
electronic  parts  (by  part  classes,  type  of  screens,  100%  or 
sampling)?  What  cost  or  cost/benefit  data  is  available?  What 
does  reject  data  indicate?  What  is  done  at  the  assembly  or 
printed  circuit  board  level?  What  is  screened  at  subsystem  or 
system  level?  What  is  distribution  of  failures  at  these  levels 
(design/parts/workmanship)?  is  there  a  failure-free  period 
required?  What  level  of  contractor  or  government  controls  the 
ESS  requirements?  Have  there  been  changes  or  revisions?  What 
costs  are  available  (equipment,  manhours,  etc.)?  How  many 
thermal  cycles  and  what  was  the  range  and  rate  of  change.  How 
has  this  been  documented?  To  what  degree  are  subcontractors 
required  to  accomplish  the  above? 
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DISCUSSION :  The  1981  guidance  pamphlet  published  by  the 

Institute  for  Environmental  Sciences  (IES)  is  a  good  baseline 
for  the  development  of  a  comprehensive  ESS  program.  The  use  of 
ESS  as  part  of  receiving  inspection  can  be  economically  justi¬ 
fied  for  some  part  types.  Engineering  judgment  must  be  made 
as  to  which  specific  screens  will  be  most  effective.  The  re¬ 
ceiving  screens  supplement  or  replace  the  screening  programs 
that  are  normally  applied  at  the  part  manufacturers.  With  the 
limited  historical  experience,  it  is  important  for  screening 
programs  to  remain  flexible.  Control  of  screening  details 
should  allow  for  adjustments  as  production  is  performed,  with 
government  control  of  some  final  result  (system  level  burn-in 
with  a  failure  free  period  might  be  such  a  criteria). 
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MANUFACTURING 


ELEMENT:  Failure  Reporting,  Analysis  and  Corrective  Action 

System  (FRACAS) 


DEFINITION :  This  element  is  a  closed-loop  activity  which 

provides  visibility  of  potential  and  observed  problems,  the 
failure  analysis  performed,  and  the  corrective  action  taken 
for  resolution.  It  is  applied  throughout  the  life  cycle  of 
the  product  to  individual  problems  and  problems  in  aggregate. 
(Factory  defects  from  production  inspection  and  test  are 
accounted  for  by  the  Quality  Program). 


SCOPE:  Two  classes  of  problems  exist  and  are  included  in  the 

corrective  action  systems.  The  first,  potential  problems, 
are  those  revealed  as  a  result  of  design  review  activities 
or  failure  modes  and  effects  analysis.  The  second,  observed 
failures,  are  failures  observed  during  the  various  phases  of 
the  manufacturing  process. 

QUESTION  TO  BE  ADDRESSED:  Is  there  appropriate  management 
support  to  allow  the  loop  to  close  effectively?  Has  the  system 
been  designed  to  collect  information  sufficiently,  accurately 
and  in  a  timely  manner?  How  were  failure  analyses  accomplished 
What  techniques  were  used?  Has  the  definition  of  "closure" 
been  agreed  to  and  precisely  defined  in  the  R  Program  Plan? 

What  management  reporting,  evaluation  and  controls  were 
implemented  to  effect  problem  closure  as  well  as  to  measure 
the  effectiveness  of  the  FRACAS? 
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RECOMMENDATIONS:  Problems  or  failures  manifest  themselves  in 


a  variety  of  ways.  These  include  catastrophic  failures  and 
out  of  tolerance  drift.  Causes  vary  from  operator  error, 
overstress,  handling  damage,  spontaneous  failure  of  piece 
parts,  aging,  degradation,  etc.  Appropriate  effort  is  required 
for  recognition,  diagnosis  and  correction  at  the  earliest  pos¬ 
sible  phase  in  a  product's  life  cycle.  Cost  in  dollars  and 
delays  increase  by  orders  of  magnitude  if  the  defect  is 
allowed  to  manifest  itself  at  higher  levels  of  assembly  or 
later  in  the  life  cycle.  Potential  problem  identification  and 
resolution  is  the  most  productive  effort  because  the  design  can 
be  changed  more  easily  before  it  is  released  for  production. 
Real  problem  identification  and  correction  have  very  tangible 


benefits/cost  visibility. 
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TEST  AND  EVALUATION 


ELEMENT:  Design  Limit  Qualification  Test 

DEFINITION :  Design  limit  qualification  testing  consists  of 

tests  to  qualify  the  design  to  the  limits  specified  in  the 
contract,  specification,  etc.  These  tests  demonstrate  that  the 
design  can  perform  satisfactorily  at  the  design  limits.  There 
is  usually  only  minimal  operating  time  required  at  each  limit. 


SCOPE:  These  design  limit  or  qualification  tests  are  usually 

drawn  from  military  standards  (e.g.,  MIL-STD-810)  and  address 
environments  (e.g.,  temperature,  altitude,  vibration,  salt, 
fog,  etc.),  compatibility  (electro-magnetic  interference), 
endurance,  and  other  factors. 


QUESTIONS  TO  BE  ADDRESSED:  These  tests  are  thought  of  by  some 
persons  as  performance  tests,  therefore  what  was  the  priority 
on  these  tests  versus  tests  in  R&M?  What  was  the  sequence  of 
these  tests?  How  did  they  relate  schedule-wise  to  the  reliability 
and  maintainability  tests?  Was  R  or  M  data  collected  from  these 
tests?  Were  there  reruns  of  sections  of  the  tests?  Were  there 
R&M  related  design  changes  addressed  that  were  incorporated 
after  these  tests?  What  process  determined  the  design  limits 
(military  standard,  mission  profile,  etc.)?  Were  failures 
from  these  tests  reported  under  the  failure  analysis/corrective 
action  program?  Are  costs  and  returns  on  investment  identifiable? 
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DISCUSSION ;  This  test  phase  is  sometimes  not  thought  ot  as  an 
R&M  element.  The  tests  examine  the  capability  of  the  design 
(normally  one  sample)  to  meet  the  performance  requirements  at 
the  design  limit  (normally  for  a  short  period  of  time  or  for  a 
short  performance  test).  There  is  usually  no  attempt  to  gain 
"hours"  to  assess  quantitative  reliability.  The  interrelationshi 
between  these  tests  and  R&M  tests  can  be  critical  to  assure  that 
the  final  design  is  adequately  tested.  The  failure  data  (and 
diagnostic  data)  from  these  tests  can  be  valuable  since  it  will 
be  some  of  the  earliest  data  available.  Design  changes  made 
later  in  the  cycle  (after  reliability  growth  test?)  should  be 
evaluated  as  to  their  impact  on  the  results  of  the  design  limit 
tests . 
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ELEMENT:  Reliability  Growth  Testing 


DEFINITION :  Reliability  growth  testing  is  a  planned,  pre¬ 

qualification,  test-analyze-and-f ix  process  in  which  equipments 
are  tested  under  actual,  simulated,  or  accelerated  environments 
to  disclose  design  deficiencies  and  defects. 

SCOPE :  This  element  includes  the  actual  growth  testing,  the 

analysis,  redesign,  and  retesting. 

QUESTIONS  TO  BE  ADDRESSED:  Was  a  reliability  growth  program 
required  by  contract,  or  self  imposed?  What  was  the  test  length 
(i.e.,  hours)  and  how  was  this  determined?  what  was  the  test 
environment  and  how  was  this  determined?  Was  there  a  reliability 
growth  prediction?  Did  the  results  track  the  prediction?  Were 
all  failures  analyzed?  How?  Were  corrective  actions  taken  on 
all  failures?  How  many  units  were  used  for  growth  testing? 

Was  some  of  the  reliability  growth  the  result  of  "burn  in"  of 
the  test  units?  What  was  the  cost  of  the  total  RGT  (test  units, 
hours,  analysis,  corrective  actions)?  How  did  the  RGT  relate 
to  the  rest  of  the  design  cycle?  When  was  it  completed  and 
all  redesign  done?  Were  there  outstanding  deficiencies?  How 
did  it  relate  to  reliability  qualification  or  demonstration 


DISCUSSION ;  Reliability  growth  testing  can  be  a  very  effective 
task  in  the  development  of  reliable  complex  equipment.  The  need 
for  this  task  depends  on  the  reliability  risk  that  is  projected 
to  remain  in  the  design  after  the  reliability  design  tasks. 

The  test  plan,  environments,  number  of  test  units,  equipment 
operating  sequence,  procedures  for  failure  analysis  and  corrective 
action,  time  phasing  with  the  rest  of  the  development  program 
are  all  critical  to  the  success  of  the  growth  test.  Management's 
interest  in  the  progress  of  the  growth  test  can  be  a  positive 
factor  in  keeping  the  growth  test  on  schedule. 
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TEST  AND  EVALUATION 


ELEMENT;  Demonstration  Testing 


DEFINITION :  This  element  addresses  the  physical  display  of 

acceptable  reliability  or  maintainability  levels. 

SCOPE ;  Demonstration  testing  is  used  in  both  DT&E  and  OT&E. 
Typical  demonstrations  include  laboratory  (stress  screening, 
CERT,  etc.)  and  operational  (maintainability  demonstration, 
qualification  testing,  etc.)  displays  of  the  system's  capability 
to  meet  some  stated  requirement. 


QUESTION  TO  BE  ADDRESSED:  Were  demonstrations  required  in  the 
contract?  What  criteria  was  used  for  demonstrations?  How  and 
by  whom  were  these  specified?  What  techniques,  specifications 
and  conditions  were  used?  Were  demonstrations  used  as  the 
sole  measure  of  acceptance?  If  so,  did  they  provide  adequate 
information  for  decisions?  How  was  data  from  demonstrations 
used?  Did  demonstration  effect  design  concepts? 


DISCUSSION:  To  some  degree  demonstrations  are  useful,  but  they 

generally  are  not  adequate  stand  alone  measures  of  system  per¬ 
formance.  Used  in  conjunction  with  other  R&M  development 
techniques,  demonstrations  have  more  potential  to  impact  system 


design.  In  the  area  of  operational  testing,  demonstration  data 
can  be  useful  if  applied  properly. 
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ELEMENT:  Operational  Test  and  Evaluation 


DEFINITION :  This  element  addresses  the  assessment  of  a  weapon 
system  under  the  use,  maintenance  and  environmental  conditions 
for  which  it  was  designed  to  be  employed. 


SCOPE:  Operational  testing  is  the  means  by  which  weapon  system 

performance  and  supportabi 1 ity  characteristics  are  evaluated. 
Weapon  system  operational  effectiveness  (performance)  and 
operational  suitability  (logistics  supportability ,  i.e., 
reliability,  maintainability,  availability,  support  costs,  etc.) 
critical  issues  are  addressed  during  test  and  evaluation  to 
determine  the  weapon  system's  capability  to  accomplish  its 
assigned  mission. 


QUESTIONS  TO  BE  ADDRESSED:  What  were  the  critical  issues  defined 
in  the  test  plan?  Did  the  critical  issues  reflect  using  and 
supporting  command  concerns?  Were  adequate  test  assets  avail¬ 
able?  Were  support  resources  (technical  data,  support  equipment, 
diagnostics,  spare  parts)  available  and  used  during  the  test? 

Were  weapon  system  deficiencies  discovered  during  test  corrected 
and  verified?  Did  information  derived  from  the  test  influence 
design  change  or  support  concept  change  decisions?  Were  thresholds 
and  goals  well  established  by  the  user,  if  not  how  were  test 
criteria  determined?  Was  there  adequate  feedback  to  the  program 
office,  contractor  and  subcontractors? 
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DISCUSSION ;  Operational  testing  should  be  used  to  aid  developers 
in  fine  tuning  a  system  to  fulfill  the  user's  needs.  Those 
areas  where  test  and  evaluation  was  used  as  a  means  to  improve 
system  performance  and  support  must  be  identified  to  illustrate 
the  benefits  of  operational  testing.  Where  test  and  evaluation 
had  no  effect  on  improving  the  system  the  reason  should  be 
identified. 
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ELEMENT:  In-Service  Assessment 

DEFINITION :  This  element  addresses  efforts  to  analyze  R&M 

problem  areas  and  develop  corrective  actions  for  systems  in 
early  operational  use. 

SCOPE:  This  element  includes  analysis  activity,  data  collec¬ 

tion,  engineering  review  activity,  and  implementation  of 
corrective  actions  to  identify  and  correct  field  R&M  problems. 

QUESTIONS  TO  BE  ADDRESSED:  Was  there  a  plan  (with  identified 
engineering  support,  data  collection  and  analysis,  etc.)  to 
identify  and  correct  R&M  problems  during  early  operational 
service?  What  data  system(s)  were  used?  What  was  involvement 
of  prime  contractor,  subcontractors,  user  personnel,  test 
activity?  How  long  was  the  assessment  period?  Was  the  system 
used  in  its  intended  manner?  How  did  R&M  of  system  change 
during  this  period?  What  was  the  cost  of  this  assessment 
program?  Was  it  adequately  funded?  If  not,  what  were  the 
shortfalls  in  cost,  schedule  and  tests? 

DISCUSSION:  Once  a  system  is  introduced  into  the  operational 

environment,  the  environment  (including  personnel)  and  full 
production  rate  can  introduce  a  new  set  of  R&M  problems 
that  must  be  identified,  analyzed,  and  corrected.  This 
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